The present invention relates to an image information displaying method and apparatus in which a motion picture is edited by operating icons or screen menus on a display screen by means of a GUI (Graphical User Interface).
In recent years, the enhancement of speed of computers and the improvement in functionality thereof have brought about the rapid progress of the variegation and improvement in functionality of software. The improvement in functionality of a production equipment for producing or arranging a TV broadcast program or video program has been advanced by use of a computer and an image editing program.
Particularly, in the recent program arrangement, the edition of image information using a motion picture editing apparatus with a random access information storage device (for example, a hard disk which is easy of handling and preservation and is low in cost) is made in lieu of an image editing method in which a video tape recorder is used to edit an image by repeating the quick feed and rewind depending a tape counter. The use of such a motion picture editing apparatus in combination with a computer and software improved in functionality and variegated has made the progress of more improvement in functionality of the motion picture editing apparatus. Here, the image information is any information which concerns an image and may include voice.
FIG. 4 shows an example of display on a screen used for an edition processing in a motion picture editing apparatus for editing a motion picture by use of a computer. With the use of a motion picture editing apparatus provided with a required edition processing software in a memory device, image information of a motion picture to be edited can be managed in a manner stored in the memory device. Reduced versions of cut images or scene images of a motion picture stored in the memory device are displayed as moving icons (M-icons) on a monitor screen to form a tree structure having cut images or scene images as its unit, thereby making it possible to perform an editing work for the motion picture. In the screen shown in FIG. 4, reference numeral 1000 denotes an editing window, numeral 1010 an editing area, numeral 1020 a material area, numerals 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, - - - moving icons of image materials or sources displayed in the material area 1020 (where K, M and N are natural numbers and K=N−2 and M=N−1 are satisfied), and numeral 1040 a tree structure which is formed by moving icons and represents the contents of edition of an edited motion picture.
The cut images, scene images, moving icons (M-icons) and so forth will now be described.
A cut image 1210 is a set of frame images obtained through one photographing by a TV camera from its start to its end, and a representative image representing a series of frame images.
A scene image 1220 is a set of plural cut images 1210 obtained for a predetermined object of photographing.
A plurality of scene images 1220 may be gathered to provide a block 1230. A plurality of blocks 1230 can be gathered on the basis of a specified theme of photographing to provide or display an edited motion picture (or program) 1240 of the specified photographing theme.
Thus, a motion picture to be edited can be arranged as image data with a hierarchical structure (or tree structure) which may be formed by several cut images 1210 selected from among a plurality of cut images 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, - - - as image materials in compliance with a specified theme of photographing, scene images 1220 each including the collection of a plurality of cut images 1210, blocks 1230 each including a group of selected scene images 1220, and a motion picture 1240 including the collection of blocks 1230 and providing a program with a title.
This tree structure is ordered and applied with data of a series of frame numbers for a motion picture, frame numbers of images at detected cut change points, cut image numbers serially applied to cut images, and sequence numbers representing the sequence of plural cut images each forming a scene image. This data or hierarchical structure information is included in control information used in the motion picture editing apparatus.
The M-icons are GUI (Graphical User Interface) parts each having a one-to-one correspondence to a cut image, that is, a representative image representing a series of frame images obtained through one photographing from its start to its end.
The M-icon is an image for the purpose of display which has a size reduced at a required ratio. The M-icons are displayed in the case where scene images or the whole or a part of a motion picture formed by a plurality of frame images are to be viewed promptly, as required. Otherwise, the M-icons may be displayed as representative ones of cut images.
The monitor screen provides a GUI screen for displaying edition information of cut images, scene images and a tree structure representative of the state of edition which is used for the edition of a motion picture. An editor uses an input device (for example, a mouse or keyboard) to select this edition information displayed on the monitor screen, thereby making it possible to give an instruction to the motion picture editing apparatus.
Thus, with the construction in which a motion picture to be edited is stored and managed as a tree structure, the motion picture editing apparatus is enabled to arrange a program and to perform an editing work with cut images or scene images taken as units.
The screen of the motion picture editing apparatus shown in FIG. 4 can be used, as a specific example, in the case where image materials 1030–1033 of an existing news program are used as materials of edition to prepare another news program (e.g. a digest version) newly.
In the display screen shown in FIG. 4, reference numeral 1050 denotes a group of edition function buttons for selecting various control functions used in an editing work, numeral 1051 a character (or appearing person) detecting button, numeral 1052 a tree structure forming button, numeral 1070 a slider for sliding the display contents of the editing area 1010 in a lateral direction, numeral 1080 a slider for sliding the display contents of the editing area 1010 in a longitudinal direction, numeral 1090 a slider for sliding the display contents of the material area 1020 in a longitudinal direction, numeral 1210 M-icons representative of cut images registered in the editing area 1010, numeral 1220 an M-icon representative of a scene image formed by one or plural cut images, numeral 1230 an M-icon formed one or more scene images, numeral 1240 an M-icon representative of a motion picture (or program) edited in compliance with a given theme, and numeral 810 a mouse pointer displayed on the GUI screen when a GUI operation is performed by a pointing device such as a mouse. Though the M-icon has thereon the corresponding reduced image for the purpose of display, the illustration or depiction of this reduced image will be omitted for avoiding the complexity, so long as there is no special necessity.
In FIG. 4, the tree structure 1040 formed by the M-icons in the editing area 1010 is such that one or plural M-icons 1210 on a first hierarchical layer representing cut images selected for edition are collected to form an M-icon 1220 representative of a scene image as a second hierarchical layer on the first hierarchical layer, one or plural M-icons 1220 representing the scene images are collected to form an M-icon 1230 representative of the collection of specified scene images as a third hierarchical layer on the second hierarchical layer, and a plurality of M-icons 1230 each representing the collection of specified scene images are finally collected to form an M-icon 1240 representing a program edited in compliance with a theme.
The number of hierarchical layers in the tree structure 1040 may be arbitrary. It is of course that an upper hierarchical layer can further be provided. Since the size of the window of the editing area 1010 is limited, there may be the case where it is not possible to display the whole of the tree structure 1040 within the editing area 1010. In order to see an undisplayed portion, the slider 1070 or 1080 is operated by use of the input device to move the display contents of the editing area 1010 in a required lateral or longitudinal direction, thereby making it possible to view the whole of the tree structure 1040.
In the material area 1020, different cut images obtained by detecting cut change points for a series of motion picture materials and dividing them at the cut change points are temporarily located as image material M-icons 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, - - - which are displayed so that it is possible to take a look at them.
At a stage in which the editing work proceeds to the invocation of images so that M-icons 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, - - - are generated and displayed, the M-icons 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, - - - are displayed in the material area 1010 at a time serial sequence in the left to right direction and the up to down direction.
Also for the M-icons 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, - - - in the material area 1020, there may be the case where the number of the M-icons is large so that they cannot be displayed in the window of the material area 1020 at once, as in the case of the editing area 1010. In that case, the slider 1090 can be operated by use of the input device to move the display contents of the material area 1020 in the longitudinal direction, thereby making it possible to view all the M-icons.
In order to perform the edition, M-icons required for edition are selected from the material area 1020 and the tree structure forming button 1052 in the function button group 1050 is then turned on. Thereby, the selected M-icon 1030 is moved or copied into the editing area 1010. Thus, the tree structure 1040 is formed to make progress on the edition.
FIG. 4 shows a state in which the progress of the edition to a certain extent is attained so that the tree structure 1040 has already been formed in the editing area 1010 and the edition is then continued to further select desired M-icons from among the M-icons 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, - - - in the material area 1020.
The M-icon displayed in the material area 1020 or the editing area 1010 is shown as one card in the illustration of the figure. However, this is a simplified form. On the actual monitor screen, each M-icon is displayed in a form having the thickness of superimposition of frame images possessed by that M-icon.
Though the edition of the motion picture is advanced, there is less the case where image materials actually required for the arrangement of a program are all image information stored in the information storage device (cut images or a series of motion pictures read as a file of one motion picture).
Namely, there may not be the case where all the plurality of M-icons 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, - - - displayed in the material area 1020 become =necessary. Only image materials associated with some M-icons selected in accordance with an object of edition or a theme of photographing are used for a program.
Also, since it is general that a plurality of scenes are included in the image materials, a search is made so that M-icons required for edition are selected from among a multiplicity of M-icons 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, - - - included in image materials or image information read from the information storage device.
The operation of the motion picture editing apparatus using the screen shown in FIG. 4 and the method of performing the edition of a motion picture while forming a tree structure by use of M-icons have been disclosed by, for example, JP-A-10-51733.
Also, JP-A-2-184181 has disclosed an example of means for detecting change points for cut images, that is, a technique of analyzing the illuminance, sound volume, color tone or the like of a motion picture to detect a portion with a remarkable change as a change point for a cut image.
In order to perform the edition of a motion picture as mentioned above, it is required for an editor that an M-icon representing a cut necessary for the program should be selected from among the plurality of M-icons 1030, 1031, 1032, 1033, - - - , 103K, 103M, 103N, displayed in the material area 1020.
One function supporting the search for the necessary M-icon is a character detecting function. The character detecting function detects an object such as a specified character or person from image materials or plural cuts or scenes. Here, this specified character, that is object is assumed as “X”.
With the character detecting function, an operator selects and inputs the combination of features possessed by the object “X” so that a search for a frame conformable to those features and its pixel region is made from image materials or plural cuts or scenes (that is, images in a section subjected to detection) by use of an image recognition technique.
JP-A-6-223179 has disclosed a method in which a cut containing a specified object therein is detected from a motion picture on the basis of the features of that character.
First, the operator designates the combination of features (for example, hair color, skin color and dress color) of a specified person of an object “X” in a series of selected M-icons (that is, a section subjected to detection where the character “X” may appear). Thereby, the motion picture editing apparatus detects an image region conformable to the designated combination of colors through an image processing so that information of the conforming image and its frame is displayed on the GUI screen.
When the frame containing the character “X” therein is detected, the editor confirms a cut which includes that frame. If this cut is appropriate as a part of the program material, the cut is selected and is inserted into the hierarchical structure 1040 of the editing area 1010 shown in FIG. 4.
In the conventional motion picture editing apparatus, a frame containing a specified image is detected from a motion picture but there is not provided a function of displaying information of the result of detection in a form which is comprehensible for an editor to use as information for edition. Also, the conventional motion picture editing apparatus is not capable of displaying a motion picture range having already been subjected to a processing for detection of a frame containing a specified character therein and a range having not yet been subjected to the detection processing so that they are displayed on a screen in a distinctive manner. Further, there is not provided a function of make a correction or change for the result of detection.